DOCUMENT RESUME 



TM 028 834 

Aubei, Judi 

Participatory Program Evaluation. A Manual for Involving 
Program Stakeholders in the Evaluation Process. 

Peace Corps, Washington, DC. Information Collection and 
Exchange Div. ; Texas State Dept, of Human Services, Austin. 
Div. of Community Resources. 

Agency for International Development (IDCA) , Washington, DC. 
R0094 

1995-06-00 
77p. 

FAO- 0801 -G- 00 - 3 034-00 

Peace Corps, Information Collection & Exchange, 1990 K 
Street, N.W., Washington, DC 
Guides - Non-Classroom (055) 

MF01/PC04 Plus Postage. 

Child Health; Data Analysis; *Data Collection; Evaluation 
Methods; Foreign Countries; *Health Programs; Mothers; 
*Planning; ^Program Evaluation; Workshops 
Africa; ^Participatory Evaluation; -^Stakeholders 
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who want to improve the health and nutrition component of maternal and child 
health programs. The methodology presented had already been used in two 
country programs in Africa, and the experiences in those program evaluations 
made this guide to participatory evaluation useful for other evaluation 
situations. Chapters 1 and 2 discuss a number of concepts that underlie a 
participatory approach to evaluation, setting the stage for the discussion of 
practical evaluation steps in chapter 3. These include: (1) preplanning 

meetings; (2) the evaluation planning workshop; (3) field data collection and 
analysis; (4) a workshop to formulate the lessons learned; (5) development of 
an action plan; and (6) the finalization and dissemination of the evaluation 
report. Chapter 4 presents some conclusions about the use of participatory 
evaluation. It stresses the importance of careful planning and the 
availability of certain human and other resources. Chapter 5 presents some 
cautions related to the use of the participatory methodology, and chapter 6 
contains a list of 19 practical references for evaluators. (SLD) 
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while at the same time fostering more horizontal, informal relationships between 
researchers and interviewees. This evaluation methodology is in keeping with 
many of the key concepts of RRA and RAP. 
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This manual is for CRS staff, particularly program managers, who want to 
improve the health and nutrition component of Maternal and Child Health 
(MCH) programs. The methodology presented here has already been used in two 
country programs, CRS/Sierra Leone and CRS/The Gambia. In both cases, 
substantive lessons were learned about what was being done well and what 
needed improvement. Further, very practical and measurable plans of action 
were developed based on these lessons that became the core of the PL 480 Title II 
planning documents. 

The strengths of the participatory approach to evaluation are several. For CRS, 
particularly in the MCH programs, this approach brings together counterparts 
and community collaborators (decision-makers and stakeholders) in the 
assessment of a program that is important to them. Involving them directly in the 
evaluation focuses the scope of work so that the issues addressed are relevant to 
the key players. 

Evaluations are not easy. The methodological approach in this manual provides a 
structure that enables managers to organize very complex programs into 
manageable components for evaluation. While this manual is designed for use in 
MCH programs, the theoretical underpinnings, evaluation methodology, data 
collection and analyst procedures presented should be useful for any staff in 
evaluating any project. 

Using the participatory methodology, with the assistance of a dynamic and 
experienced evaluation coordinator, can be a tremendously valuable learning 
experience for CRS and counterpart staff. The manual provides a step by step 
guide from the beginning to the end of an evaluation process. 
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CRS staff and counte:parts in Sierra Leone and The Gambia were very satisfied 
with the process and the results. They were pleased to learn so much about 
evaluation through very practical experience. They were surprised that so many 
useful lessons were learned. They were encouraged by the action plans that were 
developed as part of the process. And, they were motivated because of their 
"ownership" of the final recommendations and action plan. 

We in the Dakar Cluster encourage you to take the time to review the manual 
thoroughly. We hope that it will become a "well-thumbed" addition to your 
library'. 



Susan L. Hahn, Ph.D. 
Dakar Cluster Director 
November 23, 1993 
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Purpose of this manual 

This manual is intended as a tool which CRS program staff and their institutional 
collaborators (counterparts and other government and community actors) can 
use to evaluate ongoing Maternal and Child Health (MCH) programs. There are 
many ways to go about evaluating a project or program. The choice of the most 
appropriate approach largely depends on the goal & objectives of the evaluation 
as well as on the availability of human & material resources for the activity. The 
methodology described here has been found to be very effective for use in 
process evaluations in which the aim is primarily to assess the program 
implementation process. The evaluation methodology focuses on the analysis of 
program activities and strategies which have been implemented and on the 
development of "lessons learned" which can be applied in the future. Given this 
orientation, it is particularly well-suited to the evaluation or assessment of 
ongoing programs. 

The concept of a "participatory evaluation methodology/' used here, implies that 
program implementors are actively involved in all steps of the evaluation 
process. Participatory approaches are currently very popular in development 
programs but participation is only effective when the aim of the participation is 
clear and when a process to structure that participation is clearly defined. A 
participatory program evaluation must be carefully designed and coordinated by 
an evaluation coordinator. This can be an external consultant or someone from 
within the organization, depending upon where the required expertise can be 
found. 
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The involvement of program stakeholders (decision-makers and information- 
users) in the entire evaluation process is based on the assumption that this will 
contribute to greater relevance of the evaluation content and to greater accuracy 
in the evaluation findings. It is also based on the belief that program stake- 
holders can both contribute to the evaluation process and can learn from each 
other and from the methodological process itself. 

While the methodology discussed in the manual is oriented toward a structured 
and comprehensive evaluation of an ongoing program, many of the concepts and 
techniques can be used both in end-of-project evaluations and in ongoing 
program monitoring activities. 

The sequence of steps included in the methodology presented here have been 
successfully used in a number of different settings. However, this does not mean 
that the methodology is perfect. Each time the methodology has been used it has 
been modified based on the previous experiences. It is hoped that the core 
methodology will continue to evolve and be improved and that the innovations 
which you develop in your own evaluations will be shared with others. 



Organization of the manual 

This manual is intended to provide readers with practical guidelines for planning 
and conducting a participatory evaluation. For those who are planning to carry 
out a program evaluation it is suggested that you carefully read Chapters 1 
through 4 before you start planning your own evaluation. 

In Chapters 1 and 2 a number of concepts which underlie a participatory 
approach to evaluation are presented. An understanding of those concepts is 
important as a basis for the discussion of the practical steps in the evaluation 
process which are described in detail in Chapter 3. Chapter 4 includes 
conclusions regarding the use of a participatory methodology. Conducting a 
successful participatory evaluation is not an easy task. It depends upon careful 
planning and the availability of certain human and other resources. In Chapter 5 
a series of caveats related to the use of the participatory methodology are 
enumerated. For those who want to read more about qualitative data collection 
and analysis, participatory training and evaluation. Chapter 6 includes a list of 
practical references which should be easily found in North America or Europe, 
though probably more difficult to obtain elsewhere in the world. 



' David Korten, development management specialist, developed the concepts of the) earning 
process approach versus the blueprint approach to program implementation and evaluation. 
His original discussion of these concepts is in an article entitled "Community organization and 
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The Learning Process Approach 
to program implementation 

The methodological approach to evaluation presented in this manual is 
profoundly influenced by the concept of a "learning process approach" to 
program implementation.' This approach differs significantly from the 
"blueprint approach" to program implementation which is the tradition-^ and 
still most widely used approach. In the two approaches to program 
implementation, the orientation to program monitoring and evaluation is 
fundamentally different. It is important to understand the differences between 
the two and why this methodology is based on the learning process approach. 

The Blueprint Approach 

In the traditional "blueprint approach" to program planning, program objectives 
and activities are defined at the outset in a "blueprint" of how the program 
should be implemented. Based on the blueprint an implementation calendar is 
developed. The monitoring and evaluation of the plan consists primarily of 
determining the extent to which the activities and objectives are accomplished on 
time. Primarily quantitative information is collected in order to verify the 
number of activities and objectives accomplished. For example, information 
would be collected on the "number of health workers trained" and the "number 
of community meetings held" compared to the number planned. In this 
approach, there is no structured system for understanding why activities were 
accomplished or not, nor how they were carried out. Neither is there a system for 
providing feedback to the program plan in order to modify it in cases where 
changes could improve the functioning of the program. In addition, in this 
approach, program monitoring and evaluation is usually the responsibility of 
program managers and it allows them to report to their superiors on progress 
toward the blueprint goals. 

The Learning Process Approach 

An alternative approach to program planning and implementation, which has 
generally been found to be both more appropriate and effective in development 
programs, is the "learning process approach." In this approach, as in the 
blueprint approach, project objectives and activities are defined at the outset and 
an initial implementation calendar is developed. Monitoring and evaluation 
activities are concerned not only with the extent to which the planned activities 
are carried out but also with how they are being carried out. In this approach 



rural development: a learning process approach" which appeared in the Public Administration 
Review, vol. 40, pages 480-511, 1980. The concepts are also discussed in a book entitled, 
Bureaucracy and the Poor: Closing the Gap David Korten and Felipe Alfonso, Kumarian Press, 




West Haitford, 1983. 
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mechanisms are developed to help program staff to learn from both the successes 
and problems encountered in implementing the activities in order to improve the 
program in the future. For example, in a monitoring or evaluation activity it 
would be important to know not only how many community meetings were held 
but also to identify both the successful and problematic aspects of those 
meetings. This would allow program staff to identify ways to improve future 
meetings. In this approach, monitoring and evaluation activities involve the 
collection of important quantitative information but priority is given to the 
collection of qualitative information which describes the process involved in 
carrying out each type of activity. For example, information might be collected on 
the "number of health workers trained" but also on the "quality of the training 
materials," on the "feed-back from the trainees" regarding the training content 
and on "how the trainees are using what they learned" in their work with 
communities. 

Based upon the information collected, "lessons learned" are formulated which 
are fed back into the program plan. Modifications in program activities and 
strategies can continuously be made based upon the lessons learned which are 
formulated during the entire period of program implementation. In this 
approach, not only program managers but all levels of program staff are 
involved in program monitoring and evaluation. Program field staff have a 
particularly important role to play in providing their observations regarding the 
activities being implemented and their suggestions of how to improve them. 
Lessons which are developed by program staff can help program managers make 
better decisions about how to adjust the program strategy, activities and budget. 

The chart below summarizes some of the key differences between the "blueprint" 
& "learning process" approaches to program evaluation related to: the purpose 
of the evaluation; the scope of the data collection; data collection methods used; 
and responsibility for data collection. 
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' Blueprint " and " Learning Process " approaches 
to program evaluation 



Purpose of 
evaluation 



Scope of data 
collection 



Data collection 
methods 



Res 
for data 
collection 



Bluerint" approach 


" Learning Process" approach 


-to measure progress 
relative to program 
"blueprint 


-to measure quanitative 
achievements 

-to develop lessons learned to be 
integratedinto program plan 


-limited no. of variables 
related to blueprint targets 


-holistic 

-additional dimensions, questions 
emerge in the iterative learning process 


-quantitative 
objective assessment 


-qualitative and quantitative 
-"subjective" judgement 


-external evaluators and/or 
program managers 


-program stakeholders (program 
managers and progam implementors) 







As stated above, the evaluation methodology presented in this manual is based 
upon a "learning process approach" to program implementation. The practical 
implications of this approach will be spelled out in the following pages. 
Perhaps the two most important facets of the approach in the methodology are 
the involvement of program stakeholders in all steps in the evaluation process 
and the focus of the evaluation on the development of lessons learned which 
are translated into an action plan. 



Glossary of terms 

In this manual, a number of terms are used which may not be entirely familiar 
to the reader. These terms are defined here. As you come across them in the 
text, you may want to refer back to this glossary. 

Blueprint approach (to evaluation): An approach to evaluation which 

measures program accomplishments against program objectives defined 
in original program "blueprint". 

Closed questions : Questions which elicit a short, often yes or no, 
response. These questions do not usually require the respondant to 
think at length before answering. 
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Content analysis : The basic approach to data analysis in qualitative 

research. In this approach the responses of an individual or a group are 
organized according to the categories identified in the responses 
themselves through a sort of discovery process. 

Data collection : The collection of quantitative and/or qualitative information 
through the use of various techniques. 

Double-barreled questions : Questions which ask two different things at 
the same time. For example, "What do you think about the training 
and supervision you received?" Such questions should be avoided in 
interviewing. 



Evaluation questions : The questions which orient and provide a focus for the 
evaluation process. These are not the same as "interview questions". In 
other evaluation manuals these are often referred to as "evaluation 
objectives." 

Impact evaluation : A type of program evaluation which aims to determine if 
there have been changes in the target group members as a result of the 
program. In health and nutrition programs, impact evaluations usually 
measure changes in morbidity(illness), mortality (deaths) and/or 
nutritional status. 

Interview questions : The actual questions asked of interviewees during 
either individual or group interviews. 

Iterative process : A cyclical process in which, experience or knowledge is 
aquired, reviewed, modified, applied, etc., in an ongoing fashion. 

Leading questions : Questions which are formulated in such a way that 
they suggest the type of answer sought. For example, the question, 

"Would you agree that the training was well done?" suggests to the 
respondant that he/she should answer affirmatively. 

Learning Process Approach (to evaluation): An approach to evaluation which 
focuses on developing lessons for future program implementation based 
on the analysis of program accomplishments and constraints. 

Lessons learned : Based on both accomplishments and difficulties in 
program implementation identified in a program evaluation, lessons 
which are developed to improve the program in the future. 

Methodolog y: A series of defined steps which are followed in carrying out a 
given task, for example, a program planning methodology or a program 
evaluation methodology. 
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Monitoring : Systematic and ongoing documentation and analysis of the 
program activities with the goal of improving how the activities are being 
implemented. 

Objective perspective : The perspective of someone who is outside of the 
community or institution and who has a relatively unbiased view of 
community or institutional values, problems, needs, etc. 

Open-ended questions : Questions which elicit in-depth, detailed 

responses from interviewees. This type of question is extensively used 
in qualitative data collection. 

Outcome evaluation : A type of program evaluation which aims to assess 
program achievements compared to planned objectives and activities. This 
type of evaluation focuses, in quantitative terms, on how many objectives 
and activities were accomplished. 

Probing questions : Following a respondant's answer, follow-up questions 
which are used to elicit additional or more detailed information relative to 
the original answer. 

Process evaluation : A type of program evaluation which is concerned with 
how program activities were implemented, primarily in qualitative terms. 
It can, however, a.so include the collection of quantitative information. 
Process evaluations seek to determine what approaches were used, what 
problems were encountered, what strategies were successful and why. 

Program evaluation : The analysis of underlying program concepts and of the 
implementation of program activities in order to determine how to 
improve program effectiveness. Evaluations can be carried out during or 
following a program. 

Purposeful sampling : The selection of a sample of people/interviewees who 
have certain characteristics or knowledge of interest to the researcher or 
evaluator. 

Qualitative data collection : The collection of in-depth, subjective information 
which reflects "insiders" attitudes, values or priorities regarding topics of 
interest. Results consist of descriptive information which explains patterns 
or trends in beliefs, attitudes, etc. 

Quantitative data collection : The collection of succinct, objective information 
on various predetermined variables or factors in order to determine their 
frequency or magnitude. Results are presented as numbers or percentages. 

Random sampling : The selection of a sample of people/ interviewees who are 
representative of the whole population. Data collection fr om a random 
sample of people allows the researcher to generalize the study findings to 
the entire population. 

1 7 
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RAP (Rapid Assessment Procedures! : The use of primarily qualitative, semi- 
structured data collection techniques to collect information in a short 
period of time on community knowledge and practices related to health 
and nutrition. The approach was developed by anthropologists working 
in public health programs. It is similar to RRA although RAP are used 
specifically in health, and nutrition. 

RRA (Rapid Rural Appraisal) : An approach to collecting information at the 
community level which involves the use of a variety of data collection 
techniques intended to help program implementors rapidly and 
progressively learn about community knowledge and practices. The 
approach was developed in the fields of agriculture and rural develop- 
ment in the 1980's. PRA (Participatory Rural Appraisal) is a more recent 
development, based on the RRA approach, in which community members 
assume primary responsibility for collecting and analyzing information. 

Secondary data source : Information which is collected from an existing 
source such as reports, registers, files, etc. This is in contrast to infor- 
mation which is collected directly through interviews, observations, etc. 

Stakeholders : Persons who have a stake in an evaluation and who will 
potentially use the evaluation findings to make decisions regarding 
program strategies and/or implementation. 



The perspective of someone who is inside the com- 
munity or institution and whose view of values, problems and needs is 
influenced or biased by the fact that he/ she belongs to the community or 
institution. 
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Why a Participatory 
Methodology? 




In most development programs, periodic evaluations are included in program 
plans. Most program managers agree that e\ aluations are both necessary and 
important to help them make good decisions regarding how program strategies 
can be strengthened. However, evaluations often do not respond to their 
expectations. Often sophisticated evaluation methods are used which mystify the 
evaluation process and alienate program staff. A widespread problem is the 
underutilization of evaluation results. 

Program evaluations should fufill two important functions. First, evaluation 
results should provide information to program managers and funding agencies 
regarding how resources have been used, whether program objectives have been 
met and if planned activities have been carried out. Secondly, an evaluation 
should lead to the development of lessons which will help program staff improve 
program implementation in the future. Many evaluations provide information for 
accountability purposes but do not generate lessons for the future. 



Functions of 
program evaluation 



Accountability to 
managers and funding 
agencies 
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There are a multitude of ways that a program evaluation can be carried out. 
Program managers must choose the approach which best fits their needs. 
Choosing the most appropriate evaluation methodology involves consideration 
of several key issues. 

Is an outcome evaluation or a process evaluation required? 

Who should be involved in conducting the evaluation? 

Should the priority be to identify problems or to develop lessons which can be 
applied in the future? 

Let us look at some of the differences between how these three questions are 
answered in a traditional approach to evaluation and in a participatory 
approach. 



Outcome evaluation or process evaluation? 

Outcome evaluation 

The need for program evaluation originally grew out of the demands of funding 
agencies for accountability on the part of program implementors. Evaluations 
have traditionally focused on assessing the quantifiable outcomes of program 
implementation (for example, the number of training sessions conducted, or the 
number of latrines built). Outcome evaluations are widely used primarily to 
investigate how many of the planned activities were or were not carried out. 
However, program managers and funding agencies alike have increasingly 
realized, in keeping with a learning process perspective, that it is not sufficient to 
know only the numbers of activities carried out. There is increasing concern that 
evaluations should also reveal, in qualitative terms, the successes and problems 
encountered in the implementation process. 





Process evaluation 

In contrast to outcome evaluation, process evaluation focuses on the program 
implementation process in order to formulate conclusions about how things were 
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done and how they can be improved. For example, how were the various 
activities carried out? What problems were encountered in conducting the 
training sessions and how were they overcome? What are the perceptions of 
both field staff and community members regarding latrine usage? In a process 
evaluation, quantitative information can be collected on the numbers of activities 
carried out but the emphasis is on qualitative information. 

Program managers must decide whether they want to carry out an outcome 
evaluation, a process evaluation or both. The evaluation methodology presented 
in this manual is for conducting a process evaluation of a program. The proposed 
methodology is intended to produce results which respond to the information 
needs of program managers, funders and field staff regarding the 
implementation of a project or program. 



Who should be involved in conducting the evaluation? 

The traditional approach 

In the traditional approach to program evaluation, one or more outside 
evaluators are given full responsibility for conducting the evaluation including: 
defining evaluation objectives; designing the evaluation methodology; collecting 
and analyzing information; and formulating their own conclusions and 
recommendations about the program. Some evaluation experts argue that the use 
of outside evaluators assures that the evaluation will be "objective." 

In many cases, however, when responsibility for program evaluation is delegated 
to outside evaluation "specialists" they adopt a top-down, doctor-patient 
relationship with program staff. Often program staff are excluded from the 
process and the secrecy can create feelings of suspicion and stress on their part. 
The use of sophisticated data collection and analysis techniques can reinforce the 
idea that the program must depend on an evaluation expert to carry out the 
exercise. Frequently evaluation results are not easily understood by program 
personnel nor readily applicable to their work. All of these factors can contribute 
to program staff's disinterest and even alienation from an evaluation activity. 

In more traditional approaches to evaluation, evaluation "experts" often reject 
the involvement of program staff in evaluations of their own programs. 
Unfortunately, in many cases where responsibility has been delegated to experts 
to define the orientation of an evaluation, program staff often resent being 
excluded and as a result often discredit or ignore "the evaluators' findings and 
recommendations." 
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The participatory approach 

In a participatory approach to program evaluation, an evaluation coordinator, 
usually from outside of the organization, works in partnership with program 
"stakeholders" in all phases of the evaluation process. Program stakeholders are 
those individuals who have a stake in how the evaluation comes out. They are 
persons who will later make decisions and use the information generated by the 
evaluation. In a participatory approach the evaluation coordinator collaborates 
with program "stakeholders" to define the evaluation objectives, to develop the 
evaluation methodology, to collect and interpret information and to develop 
conclusions and recommendations. 

In this approach, the evaluation coordinator's role is not only to structure and 
facilitate each step in the evaluation process but also to contribute as a full 
member of the evaluation team. The role of the stakeholders is: to share their 
experiences working with the program; to participate in collecting additional 
information about program implementation; to work with the evaluation team to 
analyze both the data collected and the experiences described in order to 
formulate conclusions about the program strategy and outcomes. The 
methodology assumes that the quality of the evaluation results will be improved 
if the process elicits the subjective perspective of program implementors along 
with the more objective perspective of the outside evaluation consultant. 

A participatory evaluation is based on the assumption that the stakeholders' 
involvement will help ensure that the evaluation addresses the issues which are 
of concern to them and will increase their sense of ownership over the evaluation 
results. It also appears that stakeholder involvement leads to greater use of the 
evaluation results by program decision-makers and implementors. The 
participatory approach provides a learning experience for the program 
stakeholders who are involved. It is an opportunity for them to reinforce their 
skills in program evaluation and to increase their understanding of the program 
strategy, its strengths and weaknesses. In addition, the participatory evaluation 
process can contribute to improved communication between program staff who 
are at different levels of program implementation. 

For each evaluation exercise, program managers must decide whether the 
participatory approach, with active stakeholder involvement, is appropriate or 
not. If a participatory approach is chosen, this manual will help the program 
manager to plan and carry out the evaluation. 
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Focus on implementation problems 
or on lessons learned? 

Identifying problems 

Traditionally most evaluations have focused on identifying inadequacies and 
weaknesses in program implementation. Given the nature of development 
programs, there are always inadequacies and weaknesses. An evaluation which 
dissects and then reports on such problems is of some value, however, it usually 
leaves program staff feeling discouraged and it may not help them to know what 
to do next to improve program implementation. 

Developing lessons learned 

The participatory evaluation methodology described in this manual includes the 
identification of implementation problems but emphasizes the development of 
lessons learned based both on the problematic and successful aspects of the 
program implementation process. From beginning to end, the evaluation 
methodology addresses the question "What can we learn from what we have 
already accomplished in order to improve the program in the future?" 




principles of participatory evaluation 



1 . The involvement of program implementors in the 
evaluation process can be of great value given their experience 
with the program. 

2. The primary focus of program evaluation should not be 
on identifying problems and inadequacies but rather on 
formulating lessons learned for use in the future. 

3. The participntory evaluation process can contribute to 
improved communication between program staff at different 
levels of program implementation. 
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3teps in the 
Methodology O 



The participatory evaluation methodology consists of six phases, each consisting 
of several steps. 



I..- .- Phases of the methodology ssss - 

Phase I: Pre-planning meetings 

Phase II: Evaluation planning workshop 

Phase III: Field data collection and analysis 

Phase IV: Workshop to formulate lessons learned 

Phase V: Development of action plan 

Phase VI: Finalization and dissemination of the evaluation 
report 



The entire process is summarized in the table on the following page. Although 
the steps in the methodology are presented as a sequence from l to 19, in some 
cases the implementation of the steps resembles a spiral more than a straight line. 
For example, in Step 3 the logistical planning begins but it cannot be completed 
until Step 9 when the sample of interviewees is finalized. 
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Table 1: Steps in the participatory evaluation process 



Phase I: 

Pre-planning meetings 
(Evaluation 
Coordinating Group) 



Phase II: 

Evaluation planning 
workshop 
(Evaluation Team) 



Phase III: 

Field data collection 
and analysis 
(Fieldwork Teams) 



Phase IV: 

Workshop to formulate 
lessons learned 

(Evaluation Team) 



Phase V: 

Development of action 
plan 

(Key program 
stakeholders) 



Phase VI: 

Finalization and 
dissemination of 
evaluation report 
(Evaluation 
Coordinator and 
Evaluation 
Coordinating Group) 



Step I: 
Step 2: 
Step 3: 

Step 4: 



Step 5: 

Step 6: 
Step 7: 
Step 8: 
Step 9: 



Step 10: 
Step 11: 
Step 12: 
Step 13: 



Step 14: 
Step 15: 
Step 16: 



Step 17: 



Step 18: 
Step 19: 



Define evaluation goal & objectives 
Identify Evaluation Team members 
Plan logistical and administrative 
arrangements 

Develop visual framework of the project 



Organize stakeholders into a working 
group 

Develop evaluation questions 
Identify data collection techniques 
Develop data collection instruments 
Finalize sample of data collection sites 
and interviewees 



Orient fieldwork teams 
Conduct interviews and observations 
Analyze information collected 
Summarize fieldwork findings 



Formulate lessons learned for each 
evaluation question 
Team assessment of the evaluation 
process 

Summarize lessons learned 



Develop action plan based on 
evaluation findings 



Write evaluation report 

Distribute and discuss evaluation results 

with project collaborators 
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There are a number of different individuals and groups who are involved in the 
evaluation process. The composition and the roles of each are summarized here. 

Evaluation Coordinating Group: A small group (2-5 persons) which ensures 
the overall coordination of the evaluation from beginning to end. In Phase 
I they are responsible for all of the initial logistical and methodological 
planning. During Phase II they should assist with all logistical 
arrangements and materials. During the fieldwork, Phase III, they serve as 
team coordinators for the data collection and analysis. In Phases IV- VI 
they should provide support as needed. 

Evaluation Team: This refers to the comprehensive group of program 
stakeholders who are actively involved in Phase II to develop the 
evaluation methodology, and Phase IV to study the evaluation findings 
and to develop the lessons learned. The team will probably consist of 
between 10 and 15 people. In cases where the evaluation team is relatively 
small (10-12 people), it may be possible for all of them to participate as 
fieldwork team members. Where the evaluation team is larger, not all of 
them will be involved in Phase III fieldwork. 

Fieldwork Teams: One or more teams of stakeholders who are responsible for 
collecting and analyzing data collected during fieldwork interviews and 
observations. They are a sub-group of the Evaluation Team members. 

Evaluation Coordinator: Someone who has experience in qualitative 

evaluation methods, in group process and in participatory training, who 
designs and has primary responsibility for facilitating all steps in the 
evaluation process. 



In this chapter each of the steps in the methodology will be described and illustrated 
with examples from a participatory evaluation of a community nutrition project 
carried out in The Gambia in West Africa in 1992. The project evaluated, the 
Nutrition Education Pilot Campaign (MEPC), was implemented by the Gambian 
Food and Nutrition Association (GAFNA), a non-governmental organization. 
GAFNA is the CRS counterpart in a Title II Health and Nutrition Program. 
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Phase I: Pre-planning Meetings 



In the first phase of the evaluation process, the evaluation coordinator meets 
with the project managers and other persons who will be involved in the 
coordination of the entire evaluation activity. Through a series of meetings the 
Evaluation Coordinating Group is responsible for dealing with the first four 
steps in the process. 




Step 1: Define evaluation goal and objectives 

The initial step in the evaluation process is to define the goal and objectives of 
the evaluation. It is important that the managers of the program to be 
evaluated be involved in this task to ensure that the goal and objectives meet 
their expectations. 

One broad goal should be defined which reflects the overall aim of the 
evaluation. The wording of the goal should be concise and simple so that all 
who read it will clearly understand the purpose of the evaluation. In the 
Gambian evaluation, the Coordinating Group defined the goal of the 
evaluation as: to assess the nutrition education strategy and accomplishments in 
order to develop lessons learned for future community nutrition education activities 
and to develop a nutrition education action plan. 

Based on the evaluation goal, several evaluation objectives should be 
formulated which define the main expectations of the evaluation. The 
objectives can address both quantitative and qualitative aspects of the 
program, learning objectives for participants, and the evaluation process 
itself. In the GAFNA evaluation 4 broad objectives were defined. 
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The GAFNA Experience 



GAFNA Evaluation Objectives 

1. to assess the appropriateness of the nutrition education 
strategy 

2. to identify the accomplishments, strengths and 
weaknesses in the implementation of the nutrition 
education project activities 

3. to reinforce the knowledge and skills of project 
implementors and collaborators in conducting 
participatory program evaluations 

4. to assess the usefulness of the participatory evaluation 
methodology for evaluating community-based health 
and nutrition interventions 



The expectation that an evaluation constitute both a "learning process" and a 
"staff development" exercise is clearly reflected in the GAFNA evaluation 
goal and objectives. In Step 6, based upon the evaluation objectives defined 
here, the specific questions which the evaluation should answer will be 
defined. 



£ 



Step 2: Identify evaluation team members 



Determination of the composition of the evaluation team should be based 
first, upon the skills required to plan and conduct the evaluation and, 
secondly, upon program managers' staff development priorities. Four types of 
knowledge and skills should be represented on the team: 1) in-depth 
experience with the program to be evaluated; 2) experience with qualitative 
data collection methods; 3) team-building and group facilitation skills; 4) 
skills in planning and managing logistical arrangements for field activities. If 
the evaluation is viewed as a staff development exercise, program staff 
involved at different levels of the program should be included on the 
evaluation team. 



Table 2, shown on the following page, summarizes the types of knowledge 
and skills required amongst the team members, and their respective 
responsibilities. Team members should include: program managers; program 
field, staff; a logistics coordinator; and the evaluation coordinator. In addition 
to these team members, in some cases it will be appropriate to include other 
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persons who have collaborated with program staff to some extent, although 
they may not have been involved in all phases of program implementation, 
for example. Ministry of Health or Ministry of Agriculture staff. 



Table 2: Composition and Responsibilities of the Evaluation! earn 



r 

Knowledge/skills required Who is responsible? Responsibilities 


In-depth experience 
with the program to be 
evaluated. 


Program managers 
Program field staff 


• define evaluation questions 

• participate in data 
collection & analysis 

• participate in report 
writing 


Experience with process 
evaluations and 
qualitative data 
collection methods 


Evaluation coordinator 


• plan & coordinate 
evaluation methodology 

• participate in and supervise 
data collection & analysis 

• coordinate report writing 


Team-building and group 
facilitation skills 
during planning and 
fieldwork phases 


■Evaluation coordinator 
■Program staff members 


• develop spirit of 
collaboration and sharing 
amongst team 
members 

• facilitate group planning 
and data analysis sessions 


Planning & managing 
logistical arrangements 
for field activities & 
material resources for 
the study 




■Program staff member 
(logistics coordinator) 


• prepare budget for training 
and field work & report 
production 

• arrange logistical aspects of 
field work 

• procure materials necessary 
for the evaluation 



Program managers 



Program managers have an in-depth understanding of the program and its 
priorities. To assure that the results of the evaluation correspond with their 
information needs, program managers should be members of the evaluation 
team. Ideally, all team menbers including the program managers, should 
participate in all phases of evaluation planning and implementation. In the 
case of program managers, however, it may be impossible for them to 
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participate as full-time team members. Nevertheless, they should be involved 
in the process as much as possible so that they assume ownership of the 
evaluation. This will increase the likelihood that they will use the results of 
the evaluation. 

Program field staff 

The evaluation team should also include program field staff who have been 
directly involved in implementing program activities. The special knowledge 
which such individuals can contribute to the evaluation is an in-depth 
understanding of the dynamics of the day-to-day implementation of the 
program which program managers usually do not have. Their subjective 
insights will be invaluable to the team as the assessment of the program 
proceeds. Their perspective will also be critical to the development of lessons 
learned for the future. 

Evaluation coordinator 

The team should include an "evaluation coordinator" who has experience 
both planning and conducting process evaluations, and with qualitative data 
collection methods. In addition, he/ she must be committed to involving 
program staff in the evaluation process and have skills in team-building and 
group facilitation. Not all evaluation consultants either believe in a 
participatory approach to evaluation nor have the skills to involve staff in all 
steps in the process. It is suggested that such an individual be referred to as 
the "coordinator" rather than the "evaluator," or some other term which 
suggests his/her superior status on the team. 

The "evaluation coordinator" is responsible for coordinating all of the 
methodological aspects of the study, participating directly in the data 
collection, supervising the other team members, facilitating the analysis of the 
data, and coordinating the writing of the final report. 

Logistics Coordinator 

One or more program staff members should be chosen to coordinate the 
logistical arrangements for both the preparatory and field work phases. This 
requires someone who knows how to systematically plan activities, to 
estimate the resources required for each activity, and to assure that the 
necessary resources are available at the correct time and place. The logistics 
coordinator should be actively involved in the initial planning meetings to 
assure that he/she has a good grasp of the logistical requirements for Phases 
II through VI. 
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GAFNA Evaluation Team members 



The Evaluation Team members included individuals who 
had been involved with program implementation at 
different levels and other key program collaborators. A 
group of 22 people was identified, composed of: 

- GAFNA central and provincial project staff 

- Health education and nutrition staff from the 
Ministry of Health 

- Ministry of Agriculture field staff 

- Community health nurses 

- CRS Program Manager 

- Evaluation coordinator 



Step 3: Plan logistical and administrative arrangements 

The success of any evaluation depends in part on advanced and careful 
logistical and administrative planning. It is important to avoid the error of 
focusing on preparing the methodological aspects of an evaluation and giving 
insufficient attention to the importance of the logistical and administrative 
arrangements. 

Planning and supervising all of the logistical arrangements is time- 
consuming. As much as posssible the program manager should delegate these 
tasks to support staff so that the manager is not distracted from participating 
fully in the methodological aspects of Phases I through V. 

At this point in the evaluation process, the logistics coordinator should be 
identified and begin the logistical and administrative planning. This planning 
will progressively be completed until me field work schedule is finalized in 
Step 10. 

The logistical and administrative planning includes: choice of data collection 
sites; determination of number of fieldwork teams and members of each; 
choice of fieldwork team leaders; preparation of a budget for all materials and 
field work expenses; purchase of materials for the training, field work, report 
writing and duplication; planning lodging and meal locations for field work 
period; arrangements for vehicles, drivers and petrol; administrative 
procedures to inform provincial level authorities of the activity and to elicit 
their collaboration. The logistical coordinator may accompany one of the field 
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teams during the field work, and/ or delegate this responsibility to one person 
on each of the other teams, to ensure all logistical arrangements. 

A useful tool for scheduling these practical aspects of the evaluation is an 
"Evaluation Activity Schedule." Such a schedule can be developed in chart 
form by listing all of the specific activities or tasks which must be carried out 
during the 6 phases of the evaluation. In another column the material resources 
required for each activity can be listed, followed by the date by which the task 
must be completed. 
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Given the need to begin planning for the fieldwork as early as possible, at this 
stage there should be preliminary discussion of the regions and sites at which 
data collection will be carried out. This will allow the logistics coordinator to 
proceed with the planning for the fieldwork period. Details of the field data 
collection will be determined progressively and finalized in steps 7, 9 and 10. 

An important tool for the determination of the fieldwork sites is a map which 
includes all of the potential data collection areas. The evaluation coordinating 
group should define the criteria for the choice of data collection zones and 
sites based upon three types of considerations: programmatic; 
methodological; and logistical. 

Programmatic considerations which might influence the choice of data 
collection sites could be, for example, the location of the program activities, or 
the fact that the same activities were not carried out at all program sites. The 
methodological considerations are related mainly to the nature of sampling 
in qualitative studies, i.e. purposeful sampling (discussed below). Logistical 
considerations that would influence the choice of the data collection sights 
are numerous: the number of fieldwork teams; the distance and time to and 
between potential data collection sites; climatic and road conditions; the time 
available for the fieldwork portion of the evaluation exercise. Sites which are 
the farthest away and the most difficult and time-consuming to get to should 
not be systematically eliminated from the sample. In many evaluations there 
is a tendency to exclude sites which are further from the capital city, further 
from the tarmac and with poorer roads. This bias should be avoided as much 
as possible. 

Obviously the availability of human and material resources for the evaluation 
will influence the sample size and the choice of interview sites. The 
availability of the various team members and of the financial resources 
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necessary to ensure their involvement in the study will determine the length 
of time the evaluation can last, the number of sites that can be visited and the 
number of interviews that can be conducted. 

The purposeful sample 

As stated earlier, in a process evaluation, data collection is primarly 
qualitative. An important methodological consideration is therefore the 
nature of sampling in qualitative data collection. In a qualitative evaluation 
the data collection sites do not need to be randomly chosen as in a 
quantitative study. Rather, a purposeful sample is chosen. The choice of a 
purposeful sample involves first defining the characteristics of the sites or 
type of persons to be interviewed. Based upon these characteristics, any sites 
or persons having those characteristics can be included in the sample. 

The determination of the data collection sites can be a time-consuming 
process given the variety of considerations that must be taken into 
consideration. The proposal will probably be revised several times before it is 
finalized. It is helpful to make a simple map on flipchart paper on which the 
data collection sites can be plotted. 



Senegal 




The CxAFNA Experience 



Evaluation Data Collection Sites 



The Gambia 




Senegal 



x = Data collection site 
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Step 4: Develop visual framework of the program/project 

The evaluation team members need to have a common understanding of the 
aspects or elements of the initial program or project strategy to be examined 
in the evaluation. In this step the Evaluation Coordinating Group develops a 
visual project framework, usually in the form of a table, which defines the 
scope of the evaluation. The involvement of program managers in this task is 
of critical importance to ensure that the elements which are included and 
excluded. from the framework reflect program managers' priorities. 

The framework should contain the program or project goal, objectives and 
activities included in the strategy. In addition, it should include the 
underlying concepts or assumptions adopted in the program strategy such as 
"community management of project activities" or "participatory training 
methodologies." 

The development of the framework serves several purposes. It requires the 
coordinating group to clarify the initial program strategy and components 
and, as such, is a first step toward defining the scope of the evaluation. The 
available project documents should be used to prepare the table. The 
preparation of the framework involves a decision-rr.aking process to decide 
what should and should not be included in the framework and hence, in the 
evaluation. The framework will be an important tool for communicating with 
the other evaluation team members who will need to have a common 
understanding of the scope of the evaluation. The framework should be 
copied onto flipchart paper and/or photocopied so that during Phases II and 
IV the expanded evaluation team will have easy access to it. 





The GAFNA Experience 






The development of the framework turned out to be a 
time-consuming task. Although many project documents 
existed, some elements of the strategy were not written 
down and had to be reconstituted by those members of 
the Evaluation Coordinating Group who had been 
involved since the outset of the project. A portion of the 
framework developed by the group is presented in Table 
3, "Nutrition Education Pilot Campaign Project 
Framework." 
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One day may be required to develop the framework depending upon 
whether al! of the necessary elements are available to the group in writing. 

Table 3: A portion of the NEPC project framework 



Goal and objectives 



Project strategy 



to design and disseminate 
appropriate nutrition education 
messages as part of maternal and 
child nutrition intervention 
strategies 



Objectives: 

to investigate the effectiveness of 
different communication meth- 
ods for the dissemination of 
nutrition education 

to promote a positive change in 
mothers' knowledge, attitudes 
and practices regarding the 
nutrition of pregnant and lactat- 
ing mothers and young children 
through the diffusion of three 
messages 

to familiarize Community Health 
Nurses and village-based tradi- 
tional communicators with 
common problems related to 
maternal and child feeding 
practices and possible solutions 



Concept: 

Interactive participatory education methods 

contribute to greater learning 

Activities: 

- group discussions, role plays and cooking 
demonstrations conducted by Community 
Health Nurses 

- songs and group discussions conducted 
by traditional communicators 

- combined approach of activities con 
ducted by traditional communicators and 
Community Health Nurses 

Concepts: 

- Nutrition education content should take 
into account cultural beliefs and practices. 

- Nutrition education strategy should 
promote changes in individual knowledge 
and practices 

Activities: 

- conduct a review of available studies on 
infant feeding and maternal nutrition 



identify three priority nutrition messages 

develop three posters 

develop a nutrition education manual for 
Community Health Nurses 

conduct baseline survey in pilot villages in 
three regions 
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Phase II: Evaluation Planning Workshop 



The second phase in the participatory evaluation methodology consists of the 
"Evaluation Planning Workshop" in which all of the Evaluation Team members 
who were identified in Step 2 participate. During the workshop the team 
members are first introduced to the basic concepts of participatory program 
evaluation. Then they are involved in actually developing the methodology to 
be used in the evaluation. The workshop should be a minimum of 4 days and 
preferably 5. Steps 5 through 9 are addressed in this phase. 




Phase II: Evaluation Planning Workshop — 

Organize stakeholders into a working group 

Develop evaluation questions 

Identify data collection sources & techniques 

Develop data collection instruments 

Finalize sample of data collection sites and 
interviewees 
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The initial workshop must be carefully planned, based upon clear workshop 
objectives. Several days before the workshop begins, the evaluation coordinator 
should draft a set of objectives which should be discussed and revised as 
necessary with the program manager and/ or Evaluation Coordinating Group 
members. 
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Evaluation planning workshop objectives 

General objective: 

to involve project stakeholders in developing the 

evaluation methodology 

Specific objectives: 

1. to define three main types of program evaluation 

2. to define the function of evaluation in a "learning 
process" approach to program implementation and 
evaluation 

3. to defi ne the concept of participatory evaluation 

4. to identify the steps in a participatory evaluation 

5. to identify the main differences between quantitative 
and qualitative data collection 

6. to describe 5 qualitative data collection techniques 
which can be used in health and nutrition projects 

7. to define the evaluation questions for the NEPC 

8. to identify from whom/what source information 
should be collected for each evaluation objective 

9. to identify the most appropriate data collection 
techniques for each evaluation objective 

10. to develop data collection instruments 



Based upon the training objectives, detailed written training designs must be 
developed for each of the sessions. This task is primarily the responsibility of the 
Evaluation Coordinator, who should be experienced in designing participatory 
training, although other Evaluation Coordinating Group members or 
experienced trainers/ adult educators may also be involved. The workshop 
sessions should be highly participative but at the same time carefully structured 
so that the necessary tasks are accomplished as planned. 

Table 3, on the following page, shows the GAFNA workshop schedule. The full 
team participated in the first 3 days of the workshop; the last day involved only 
the field team members. 
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Schedule for Evaluation Planning Workshop 



The GAFNA Experience 



Day 1 



8:30 - 12:30 
Introduction 
Presentations 

Workshop goals and 
objectives ■ 

Concepts of program 
evaluation 

3 types of evaluations 

Learning process 
approach to program 
implementation and 
evaluation 



Day 2 



Nutrition education Developing data 



project framework 

Developing 

evaluation 

questions 



collection 

instruments 



Day 4 



Verbal and non- 
verbal 

communication 

behaviors 

Principles and steps 
of in-depth 
interviewing 

Types of questions 



14:00 - 16:30 

Participatory 

evaluation 

Overview of RRA, 
PRA, and RAP 

Qualitative and 
quantitative methods 

Steps in participatory 
evaluation 



Defining data 
collection sources 
and techniques 



Developing data 

collection 

instruments 



Active listening: 
exercise in small 
groups 

Principles of note- 
taking 
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Step 5: Organize project stakeholders into an effective team 

The evaluation team members have been identified and the expectation is 
that they will work together to carry out the various tasks associated with the 
evaluation. However, teams of people do not necessarily work effectively 
together. An initial step in Phase II is to begin to develop a sense of team 
membership and mission. The evaluation coordinator is responsible for 
designing and facilitating a series of team-building exercises at this stage in 
the evaluation process. However, an ongoing effort must be made during the 
entire evaluation process to encourage a spirit of openness and collaboration 
between the team members. 

A series of workshop exercises used in this step prepare team members to 
carry out the tasks expected of them. The participants should be introduced to 
the concept of participatory evaluation and to the role of each individual on 
the evaluation team. The rationale for the involvement of all levels of 
program implementors in the evaluation process in terms of what they can 
both contribute and learn should be discussed. Tine notion that the evaluation 
is a "learning process" in which the contribution of all team members is 
important in generating lessons for future programs should also be stressed. 
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"Demystifying evaluation" or "Why am I here?" 

At the outset of the workshop, many of the participants felt 
uneasy and unclear about their role on the evaluation team. 
They assumed that in the evaluation outside "experts" 
would be scrutinizing their work and didn't understand 
what their role could possibly be. In the workshop, special 
effort was made to demystify the concept of "program 
evaluation" and to convince the participants that they each 
had an important role to play in the process. By the end of 
the second day, it was clear to all of the participants what 
was expected of the team and of each team member. 



Other concepts related to program evaluation which the participants need to 
understand in order to develop the evaluation methodology include: the 
different types of program evaluation; basic concepts and methods in RRA, 
RAP and PRA as contrasted with traditional approaches to data collection; 
and the steps in the participatory evaluation methodology. 
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Step 6: Develop evaluation questions 

Based upon the background concepts of program evaluation presented to the 
evaluation team members in Step 4, here they will begin developing their 
own evaluation plan. In this step the question to be answered is: What do you 
want to find out? The involvement of the project stakeholders in this task is 
important for several reasons: 

1) to ensure that the evaluation responds to the concerns of program 
managers and field staff; 

2) to promote program stakeholder's sense of ownership of the 
evaluation; and 

3) to provide the evaluation coordinator with a clear understanding of 
staff priorities for the evaluation. 

A preliminary step in this task is the presentation of the project framework 
(developed in Step 4). The coordinator must be sure that the elements of the 
framework are understood by all team members. Once this is accomplished, 
the team members should be divided into small groups of 3-6 persons to 
develop the evaluation questions. Each group can be assigned to work on one 
portion of the f jamework. For example, one group could work on the project 
"nutrition education" activities while another could deal with the activities 
carried out with "mothers' management committees." 



The GAFNA Experience 

The program stakeholders were divided into four groups 
corresponding to the four components of the project 
framework: 

!) project management and monitoring; 

2) nutrition messages and materials; 

3) nutrition education sessions at the community level; 

4) training. 
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The task of each of the groups is to define specifically what information 
should be collected on their aspect of the program. For example, in terms of 
the training activities, what is important to find out in the evaluation, both 
quantitatively and qualitatively, regarding how those activities were carried 
out. The evaluation questions developed by each group should be recorded in 
a Data Collection Planning Chart (see page 36). It is important that each 
group has a large chart which is hung on the wall. This will help all group 
members to follow the work and to actively participate. 





It is important to point out that the "evaluation questions" define the 
information which the evaluation seeks to collect. However, the formulation 
of these questions is not necessarily that which will be used in actually 
interviewing program collaborators. In Step 8, when the interview guides are 
written, the actual "interview questions" to be asked will be developed. 
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Step 7: Identify data collection sources and techniques 



Once the evaluation questions are defined, the team must decide: 



1) whether the required information is quantitative and/or 
qualitative; 



2) from whom or what source the information should be collected; 

3) what data collection techniques would be most appropriate. 

Each of these decisions must be made based on the evaluation questions 
developed in Step 6. 



In a process evaluation such as this, data collection is primarily qualitative. 
As an introduction to this step, the characteristics of a qualitative approach to 
data collection, as contrasted with a quantitative approach, should be 
presented to the team. The differences should be explained in terms of: 
sample size; sample type; data collection techniques; approach to data 
analysis; and the relationship between interviewer and interviewees. 



Contrasting characteristics of quantitative and qualitative 
data collection methods 



Quantitative methods 



Qualitative methods 



Sample size; 

• large sample 

TypAo f -sai^ pte .: 

• random sample 

Data collection technique: 

• individually administered 
questionnaire 

Batajflajysia 

• information collected is classified 
according to predetermined 
categories 

Re ; onship between interviewer 
interviewees: 

• formal, structured, distant 



• small sample 

• purposeful sample 

• individual or group interview, 
observation 



• information collected is 
classified according to the 
categories of responses which 
are identified in the data itself 

• less formal, semi-structured, 
attempt to get as close to the 
community as possible 
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In evaluations of community programs, numerous data collection techniques can 
be used. In the RAF, RRA, and PRA methodologies developed in the last several 
years, more than 20 different data collection techniques have been experimented 
with. Many of them can potentially be used in program evaluations. (See 
additional references in Chapter 6.) In this manual, however, only 5 techniques 
will be discussed, those which are more frequently used in program evaluations, 
namely: in-depth individual interviews; group interviews; key informant 
interviews; observations; and secondary data sources. These techniques can be 
used to collect information both at the community and institutional levels. In the 
workshop, each of the techniques should be described. Participants can be asked 
to identify the strengths and weaknesses, and possible uses of each. 



i f 
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~ Key data collection techniques 

In-depth individual interview: A semi-structi. red interview 
using a flexible interview guide consisting of mainly open- 
ended questions. The aim is to collect detailed infonnation on 
the individual's opinions, beliefs and aspirations. 

Key informant interview (a special type of individual 
interview): A "key informant" is someone who has extensive 
experience and knowledge on a topic of interest to the 
evaluation. At the community level, often key informants are 
community leaders. The interviewer must develop a 
relationship of confidence with the informant before he/ she 
will share experiences, and insights. 

Group interview: Similar to the individual interview but 
conducted with a small group of people with similar 
characteristics and experience. Allows group members to 
discuss their opinions and attitudes with others. Interviewer 
can determine the trends in group beliefs and knowledge 
based on the responses of the group members. 

Observation: While an activity is going on, an observer 
records what he/ she sees either using a checklist or by taking 
descriptive notes. The observation can include information on 
the setting (the actors, context, surroundings); the actions and 
behavior of the actors; and what people say, including direct 
quotations. 



Secondary data sources: Reports and other written 
documents which provide information on the activities 
planned and carried out. 
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With the above introductory information, participants should then be able to 
decide, for each of the evaluation questions: 

1) whether the information to be collected is quantitative and/or 

qualitative; 

2) from whom or what source the information should be collected; 

3) what data collection techniques should be used. 

Regarding the "data collection source", it is important that information 
collection not be limited to community-level data gathering. Even though the 
program evaluated may be community-based, in many cases it will be 
important to collect information from institutional personnel who are 
involved in the program in order to adequately answer the evaluation 
questions. 

For this task, the participants should work in the same small groups to which 
they were assigned in Step 6. As in Step 6, their work should be recorded in a 
large Data Collection Planning Chart. Examples from the work on the 
"Training" component of the GAFNA project are seen below. The teams will 
probably require 3 to 4 hours to complete this task. 



The GAFNA Experience 



Data collection planning chart 

Program component; 


Evaluation questions Quantitative ! Information 

&/or | sources (where or 

qualitative j from whom) 
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collection 
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Step 8. Develop data collection instruments 



Based upon the evaluation questions developed in Step 6 and the choice of 
data collection techniques in Step 7, team members will develop draft data 
collection instruments for the interviews and observations to be carried out. 
In this step team members will continue to work in the same groups as in 
Steps 6 and 7. For each evaluation question, and for each category of persons 
from whom they have decided information should be collected (as recorded 
in the third column of the Data Collection Planning Chart) the team develops 
one or more interview questions or observation elements. 



When each team has completed this task, the questions and observation 
elements should be compiled into one interview and observation guide for 
each category of interviewee. For example, there would be one guide for 
women's management committees, and one for nurses. The Evaluation 
Coordinator should provide the team with both an explanation of how the 
interview guide and observation guide will be used as well as guidelines on 
how to develop each of these types of instruments. Developing good data 
collection instruments is not an easy task and it is not expected that team 
members will produce perfect instruments in a short period of time. For this 
reason, following preparation of the draft instruments by the team members, 
time should be allowed for the evaluation coordinator and one or more 
members of the evaluation coordinating group to review and revise the 
instruments. 



An interview guide is a set of mainly open-ended questions used either with 
an individual or group to collect detailed, primarily qualitative information. It 
is important to point out that the in-depth interview guide is quite different 
from a traditional questionnaire. A questionnaire is used in a very structured 
and identical fashion with each interviewee. The in-depth interview guide, on 
the other hand, is a more flexible tool which "guides" the interviewer but 
which allows him/her to adapt or modify the questions, to some extent, as 
the interview proceeds. For the purposes of the evaluation, one interview 
guide must be developed for each type of group to be interviewed, for 
example, "project supervisors" and "community health nurses." While the 
guides for each of the types of groups to be interviewed will differ, many of 
the questions will be the same, given that certain evaluation questions will 
apply to more than one category of interviewees. 

The necessary ingredients for development of an interview guide include the 
evaluation questions to be answered through the interview and knowledge of 
how to formulate good questions. In a process evaluation, the interest is 
primarily in collecting in-depth information on people's attitudes, opinions, 
and knowledge Therefore, primarily open-ended questions are required to 
elicit in-depth qualitative information. In addition, the formulation of the 

4 D 
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questions should follow these criteria: 

They should be clearly and simply worded. 

The formulation of the question should not suggest the answer. 

They should not be "double-barreled", ie, asking two things in the same 
question. 

They should not be formulated in the negative. 

The questions should be carefully sequenced with easier questions preceding 
the more difficult ones, and the less intimate ones coming before those which 
ask for more personal answers. The interview guide may contain questions 
which require both quantitative and qualitative responses. 



A portion of the interview guide for Community 
Health Nurses 

1. What was your role in the Nutrition Education Pilot 
Campaign (NEPC)? 

2. What ivas the role of the Mothers' Committee in the 

program? 

3. To what extent did they assume that role ? 

4. Did you receive your fuel subsidy? 

5. Was the fuel given adequate for carrying out your NEPC 

activities? 

6. What was your role in monitoring the NEPC activities? 

7. Were you trained on how to monitor the NEPC 

activities? 

8. What information did you collect in the monitoring? 

9. How frequently did you monitor the activities? 

10. Did you encounter any obstacles in monitoring the 

activities? 

II What did you do ivith the information you collected? 

[ 
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In in-depth interviewing, it is preferrable that notes be taken on sheets of plain 
paper, because the length of the responses cannot be anticipated, and also 
because the notes of responses should not be limited by space on an interview 
guide. The interview guides can be typed double-space to facilitate reading but 
space does not need to be left on the guides to record interviewees' answers. 

In qualitative data collection, the pretesting of the interview guide is less 
important than in cases where a structured questionnaire is used. This is 
because the questions can and should be modified, if necessary, during the 
data collection process. If time is available, members of the evaluation team 
can informally test the guides, and modifications can be made before they are 
copied. 

An observation guide is a tool which helps the observer structure his/her 
observation of certain aspects of an activity which are of interest . r the 
evaluation. While the activity is being carried out, the guide is used by the 
observer, as discretely as possible, to record key information on the content 
and/or process employed. Development of an observation guide requires 
understanding of the evaluation question to be answered and identification of 
the key aspects of the activity to be analyzed. 



Observation guide for performance 
by traditional communicator 



1 


Adequate 


Rather 

Adequate 


Not 

Adequate 


Remarks 


Message 

content 










Flow/Sequence 










Participant 

involvement 










Discussion 

. . 










Role play 










Songs 





















In the GAFNA evaluation only one day was allotted for the development of 
the data collection instruments and this turned out to be inadequate. One day 
is probably sufficient for the evaluation team members to develop the draft 
instruments, but an additional one or two days is required for the revision of 
instruments and for producing multiple copies which are ready for use by the 
fieldwork teams. 
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Step 9: Finalize sample of data collection sites and interviewees 



Final decisions must be made regarding the data collection sites, the sample 
of persons who will be either observed or interviewed at each site, and the 
schedule for the data collection by site. Due to the complexity of determining 
the data collection sample, this task is mainly the responsibility of the 
Evaluation Coordinating Group. In Step 3 the data collection zones and sites 
were determined. In Step 7 the types of persons to be interviewed and 
observed were identified. In this step final decisions must be made regarding 
the characteristics and number of each type of interviewee per data collection 
site as well as the detailed scheduling of the site visits, i.e., date and time. 
Based on the decisions made in Steps 3 and 7 regarding the data collection 
sites and types of interviewees, here final plans are made regarding the data 
collection sample. The "Data Collection Planning Worksheet" , shown below, 
can be a useful tool for deciding what types of data collection will take place 
at each site. 



- Data Collection Planning Worksheet — 

Region: 

Number of groups/individuals to interview or observe per site: 



Data collection 
sites 



Kfuxuatt. 



B**tLaAa 



community 
health nurses 



mothers 



traditional 

communicators 




Once this worksheet is completed, another table should be prepared which 
indicates for each site the dates and approximate time that each type of 
group /individual will be interviewed or observed. 

Decisions regarding the evaluation sample should be prepared by the 
coordinating group, and their proposal for data collection sites and 
interviewees presented to the entire evaluation team at this stage in the 
workshop. 
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Planning for group interviews 

The number of persons who participate in each group interview should be 
from 6 to 10. Groups of this size are desirable for two reasons. The small 
group allows all participants to contribute their ideas to the discussion and is 
much easier for the interviewer to facilitate than is a group with more than 10 
participants. 

Based upon the characteristics defined for each type of interviewee, the team 
should decide upon the best process for choosing specific group members. It is 
important that whoever is responsible for identifying interviewees clearly 
understand the characteristics defined for each type of group and the number of 
interviewees per group. Scheduling the interviews should take into account the 
time required to conduct each interview (approximately 1 '/ 2 hrs). Generally a 
maximum of two group interviews can be conducted per day per team. 




Phase III: Field Data Collection and Analysis 



Based upon the elements of the evaluation methodology developed in Phase n, 
the third phase consists of the data collection and analysis carried out in the field. 
As explained earlier, not all evaluation team members will participate in this 
phase unless the evaluation team is small (maximum 10-12 persons). If all 
evaluation team members participate in the fieldwork, one or two days should be 
added onto the Phase II planning workshop for the Step 10 orientation. Whatever 
the number of persons who participate in Phase III, they should be divided into 
several fieldwork teams, probably with 4-6 members each. Fieldwork team 
members will be involved in all four steps in this phase. 



Phase III: Field Data Collection and Analysis 



Step 10: Orient fieldwork teams 
Step 11: Conduct interviews and observations 
Step 12: Analyze information collected 
Step 13: Summarize fieldwork findings 
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Step 10: Orient fieldwork teams 

Prior to beginning the fieldwork, an essential preliminary step is the 
orientation of the fieldwork team members. The orientation, which should 
last one or, preferably, two days, should focus on several key topics: logistical 
arrangements for the field work phase; roles and expectations of team leaders; 
roles and expectations of team members; in-depth interviewing skills; note- 
taking; and the analysis of qualitative data. During the orientation, priority 
should be given to the development of team members' in-depth interviewing 
skills. 

Logistics and expectations of team members 

It is important that the fieldwork team members be informed of the logistical 
arrangements which have been made for the fieldwork period. Each team 
should be informed of: the travel schedule for each day in the field; the 
lodging and meal arrangements; and the sites where they would be 
conducting interviews. Most of the logistical and administrative planning for 
the fieldwork should have been done ahead of time but the final details can 
be discussed at this time. 
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Each fieldwork team has a leader. The role of the team leaders must be 
explained to the leaders themselves, as weil as to the other team members. 
Team leaders' responsibilities can include: to assure that necessary logistical 
arrangements are made; to develop a sense of team spirit and cooperation 
amongst the team members; to encourage the participation of all team 
members and respect for each member's contribution; to delegate 
responsibility for data collection and analysis according to individual ability 
and interest; to reinforce the interviewing techniques presented during the 
orientation; to facilitate collective analysis of the data collected on a daily basis; 
to ensure that by the end of each day the synthesis of the data collected that day 
is completed in writing; to ensure that by the end of the field work period 
information has been collected relative to all of the evaluation objectives. 

Team members must also understand the expectations of them during the 
fieldwork phase. They include: to contribute to the well-being of the team in 
all possible ways; to accomplish the team task in the field; to participate in 
conducting interviews; and to participate on a daily basis in analyzing the 
data collected. 

In-depth interviewing skills 

The quality of information collected in both individual and group interviews 
depends to a great extent on team members' in-depth interviewing abilities. 
During the workshop, the attitudes, knowdedge and skills required to conduct 
effective in-depth interviews should be addresed. Those same skills should be 
continually reinforced by the team leaders during the fieldwork itself. 

What makes a good interviewer? ; 

- respect for group participants' ideas, 
cultural values, traditional ideas 

- horizontal relationship between 
interviewer and interviewees 

- commitment to "learn" from interviewees 
rather than to "teach" 

- familiarity with evaluation questions and 
interview guides for each type of 
interviewee 

- principles of group dynamics 

- non-verbal behaviors which encourage 
and discourage interviewees 

- steps in the interviewing process 

- in-depth questioning strategies 

- active listening 

- note-taking 

- small group facilitation 



Attitudes: 



Knowledge: 



Skills: 
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All of the above topics should be addressed during the orientation. As many 
exercises as possible should be carried out which allow team members to both 
observe and practice using interviewing skills. Participatory learning 
activities, including role plays and exercises in pairs or triads, can be 
designed for each of these topics. 

Role plays, such as the examples shown below, can be used to help 
participants identify effective and ineffective approaches to interviewing. 
(Remember that role plays are more likely to accomplish their objectives if 
they are practiced ahead of time.) 

r, ' Sample role plays ■■■ " — y 

TM i nef fective, i nt erviewer 

An interview on infant feeding practices is carried 
out with a mother using a structured questionnaire. 

The interviewer demonstrates both verbally and 
non-verbally a lack of interest and disapproval of 
the interviewee's responses. He is not attentive 
when the mother is speaking, he verbalizes his 
impatience with her and abruptly interrupts her 
several times when she gives a lengthy answer or 
when her responses are not what he anticipates. 

The effectiKeintfiEOfiflg 

A group interview is conducted with 4 mothers 
regarding their beliefs and practices related to 
diarrhoea. The interviewer demonstrates, both 
verbally and non-verbally, her undivided attention 
and interest in the responses of each of the mothers. 

She encourage^ all of the group members to 
participate, asks good follow-up or "probing" 
questions to get the interviewees to expand upon 
their initial responses, and she accepts all of their 
responses with the same degree of openness and 
enthusiasm. 

— — — — — 1 

Following the dramatization of each role play, participants can be asked to 
assess the interviewer's approach. The group can identify strengths and 
weaknesses in the interviewer's attitude and skills, and develop a list of 
"do's” and "don't’s" for their own interviews. 
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Questioning strategies 

In interviewing, the primary tool for collecting information is the question. 
Team members must recognize that there are different kinds of questions, that 
different types of questions elicit different types of responses, and that certain 
types are more appropriate for in-depth interviewing than others. 

Closed questions should be used to a limited extent. 

Example: Did you like the nutrition education program? 

Leading questions should be avoided. 

Example: Would you agree that the traditional communicators were 
popular with the women? 

Double-barreled questions should be avoided 

Example: Did the women's committee successfully organize the growth 
monitoring and the role play sessions? 

Open-ended, probing and follow-up questions should be 

extensively used. 

Examples: What were your impressions of the activities carried out by 
the traditional communicators? 

Why do you say that they were effective? 

What do the others in the group think about what Mrs. Diallo has said? 

The interview guides for each category of interviewees are the basis for 
conducting the individual and group interviews. Unlike a questionnaire 
which should be rigidly followed, the interview guide is used in a more 
flexible manner. If the interviewee does not adequately understand one of the 
questions included in the guide, the interviewer can reformulate the question 
until the meaning of the question is clear. In many cases, in addition to asking 
the questions included in the guide, the interviewer will need to ask other 
probing and follow-up questions to assure that precise and detailed 
information is obtained from the interviewees. The aim of the questions is to 
collect information which answers the evaluation questions. The interview 
guide is a tool which the interviewer must use in a flexible and creative way 
in order to accomplish that objective. 
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Principles of note-taking 



Almost as important as knowing which types of questions to ask is knowing 
how to effectively record the responses given. The following principles of 
note-taking should be respected. 



1. Notes should be recorded in the first person. 

2. Key words and ideas should be recorded. 

3. Original, descriptive phrases or sayings should be recorded word 
for-word as quotations. 

4. Information should be recorded exactly as it is heard and not 
"filtered" based on interviewers' ideas or values. 



5. As many notes should be taken as possible. 

6. In group interviews, the different opinions in the group should be 
recorded 



Materials to be prepared for each data collection team 

copies of the evaluation questions 
copies of interview & observation guides 
large writing pads or plain white writing paper 
pens 

• file folders to organize notes by category of 
interviewees 

paper clips and/ or stapler and staples 



Step 11: Conduct interviews and observations 

Once the orientation session is completed, the fieldwork phase of the 
evaluation can begin. In fact, it is important that the orientation be conducted 
immediately before the data collection phase so that the team members do not 
forget the skills they have learned. 






The daily data collection schedule 

In most cases, the majority of the data collection will consist of interviews. 
There are several considerations in planning the interviewing schedule. 

1) Data collected in each interview should be analyzed the same day 
that it is collected. 

2) Between 2 and 4 hours is required to analyze the data collected in a 
group interview and somewhat less for an individual interview. 

3) It is easier for the team to conduct the interviews in the morning in 
order to be able to complete the data analysis by the end of the 
same day. However, the time of day at which the interviews will be 
held should be decided in discussion with persons who are familiar 
with the daily schedule of the target group members and/or 
through discussion with target group members themselves. 

Experience has shown that it is usually feasible for a team, composed of one 
interviewer and one or two note-takers, to conduct one to two group 
interviews or two to three individual interviews per day. Planning the daily 
fieldwork schedule should take into account time for: travel to the data 
collection site; greeting institutional or community leaders; gathering 
interviewees together or waiting for them to arrive; conducting the 
interviews; traveling to the lodging site; meals; rest time; and data analysis. 
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Preliminary contacts with institutional or community leaders 

Once the schedule for conducting the interviews is set, someone should travel 
to each interview site and meet with the institutional or community leaders. 
The purpose of the upcoming team visit should be explained to them and 
their collaboration in organizing the interviews elicited. Specifically, their 
help will be needed in identifying the potential interviewees and inviting 
them to participate. The designated person should be provided with detailed 
information about: the characteristics and number of individuals or groups to 
be interviewed; the date and time of each interview. Where group interviews 
are to be conducted, it is extremely important to explain the rationale for 
small groups, 6 to 10 persons. It is often assumed that the visiting team will be 
impressed if the groups are larger. Giving the contact p^^sons at the sites clear 
information about the team's objectives and expectations will help ensure that 
when the team arrives at the interview site, they will find the appropriate 
type and number of persons ready to be interviewed. 

Conducting group interviews 

The following steps and practices should be followed iwhen conducting 

group interviews: 

1. Before the interview starts 

Choose a quiet, secluded spot 

Verify the characteristics of group participants 

Limit the number of participants to approximately 10 

Seat participants in a circle 

Sit at the participant's level 

2. Introduction to the interview 

Greet the participants 

Explain the purpose of the interview 

Assure them of the confidentiality of their responses 

Encourage the participation of all group members 

^facilitate. th e , group discus s ion 

Conduct interview in local language 
Use interview guide to structure the interview 
Interview should last 1 to 1 '/ 2 hours 
Take as many notes as possible 

4. Ending the interview 

Review interview guide 
Thank the participants 
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data collection is an iterative or spiral-like learning process. In other words, in 
the analysis of the information collected in one interview certain aspects of 
the information may be unclear, others may suggest additional facets of the 
topic which were not initially included in the evaluation questions. In such 
cases, the insights from the analysis of one interview will suggest 
modifications which should be made in the content of subsequent interviews. 
A second reason why immediate analysis of the information collected is 
important is that while extensive notes should have been taken, the facilitator 
and note-takers usually retain additional information and insights about the 
interviewees' knowledge and attitudes which do not appear in the notes. 
Therefore, the sooner the data analysis takes place, the more easily this 
information can be retrieved. 

The analysis of qualitative information is considerably more difficult than 
that of quantitative data. In qualitative data collection there are guidelines, 
but there are no set formulas for the analysis of the data. The process of 
analyzing qualitative data can be fascinating but it is also very demanding in 
terms of the intellectual energy and creativity which it requires. The data 
analysis processes should be facilitated by someone with skills both in 
qualitative data analysis and in the facilitation of team work. In this regard, 
the role of the team leader as facilitator of the data analysis process is critical 
and he/ she must be carefully chosen. 

The main technique for analyzing qualitative data is content analysis. Through 
content analysis of the information collected conclusions are formulated for 
each of the evaluation questions. The analysis process involves identifying the 
categories of responses in the raw data. 

Daily data analysis 

The daily data analysis process is structured around the interview questions 
asked of each category of interviewee. For example, if one day the team 
conducts individual interviews with two nurses and one group interview 
with mothers, the two sets of data (nurses and mothers) will be analyzed 
seperately based on the questions in the two interview guides. 

During the analysis the team will often want to refer back to the evaluation 
questions to clarify the objective of the interview questions. The entire team 
should participate in the analysis sessions in which a simplified approach to 
the content analysis of the interview information collected is used. 

1. Re-read the interview questions . One at a time the interview 
questions are read to the group. This allows the team members to 
recall the focus of each interview question. 
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2. Read the interview notes . The note-takers read aloud the responses 
found in the notes for each question. If there are more than one set 
of notes, all should be read. 

3. Discuss the responses . The team leader asks the group to examine 
the information taken down in the notes, and also other comments 
made by the interviewees that may not have been written down, to 
clarify exactly what the interviewees were saying. 

4. Categorize the responses and summarize findings . Together the 
group identifies the categories of responses in the information 
collected and summarizes the findings in a concise fashion. The 
example below illustrates a summary of the findings for one 
interview question. 

= Summary of findings for an — — 

interview question 

Question:What did women think about the use of songs in the 
nutrition education program? 

For example, perhaps all of the interviewees agreed that 
the use of songs was beneficial while a few stated that 
they were repetitive. The finding for this evaluation 
question might read: 

"The women interviewed unanimously reported that the 
use of eongs was both enjoyable and an easy way for 
them to understand and to remember the nutrition 
messages. A few of the interviewees stated that the songs 
were too repetitive and that new songs should 
continuously be introduced." 

L - - ; - - -■■■■■ J 




As the example illustrates, the summary should indicate the trends 
in the information collected in terms of whether the attitudes or 
ideas expressed were shared by all interviewees, the majority, half, 
a minority or few of them. While the objective is not to quantify the 
different types of answers, the trends should be reported. 

5. Identify unclear or missing information . The last step in the 

discussion of ach interview question is for the group to determine 
whether there is any missing or unclear information that should be 
further investigated in subsequent interviews. Each of the team 
members should take note of supplementary questions which the 
team decides should be added to the interview guide. 
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Synthesis of data analysis 

At the conclusion of the field work phase, at least one day must be scheduled 
for each of the field teams to summarize its findings. In a working session in 
which all members of the fieldwork team participate, findings must be 
synthesized for each evaluation question. In this session the evaluation 
questions are read one at a time, the findings from the different groups of 
interviewees which are relevant to each question are read from the daily 
analyses (prepared in Step 4 of daily analysis process) and the team findings 
are written for each evaluation question. 

Step 13: Summarize fieldwork findings 

At the conclusion of the data collection phase, the findings from the different 
fieldwork teams must be summarized. For each evaluation question, the 
findings of all of the teams must be integrated into one set of evaluation 
findings. This task should be carried out by the team leaders in collaboration 
with one of the members of each of the fieldwork teams. It is strongly 
recommended that this task not be carried out with all of the fieldwork team 
members. The participation of all team members in this task would make it 
very time-consuming. 



In this step, for each evaluation question two representatives of each of the 
fieldwork teams should present their team's findings. Based upon the 
findings of the various team&, a summary of the findings relative to each 
evaluation question must be developed. The summaries should be concise 
and clearly worded so that they can easily be understood without referring 
back to the original evaluation questions. This approach will greatly facilitate 
the discussion of the findings by the evaluation team in Step 14. 



Examples of findings for two evaluation questions: 

Question: To what extent did the community contributed to 
the project? 

In human resource terms the communities have made a 
considerable contribution to the project: the participation of 
the Traditional Communicators and Management 
Committees; the attendance of the mothers at the sessions. 

" continued = ” * 
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They have not, however, made any financial contribution to the 
project. This arrangement may contribute to dependency on 
"outsiders" rather than to a strategy wliich is sustainable within 
the community. 

Question: To what extent was the project conducted in 
collaboration with the Ministry of Health? 

The ministry staff interviewed stated that the Nutrition Education 
project is in keeping with the Ministry of Health's health/nutrition 
concerns and objectives. However, one of the Regional Health 
Teams stated that the program was "running parallel" to the 
Ministry of Health programs rather than as an integrated 
component of their program. This Regional Health Team stated 
that they wish to be more involved in the initial planning of such 
projects and that periodic coordination meetings should be 
organized by project staff at their level. 



The findings for each of the evaluation questions should be typed and multiple 
copies prepared for the Phase IV workshop. 




Phase IV: Workshop to Formulate Lessons Learned 



In most program evaluations, the external evaluators are responsible for 
formulating recommendations based on evaluation findings. In keeping with the 
concept of a participatory evaluation, in this methodology the Evaluation Team 
members are responsible for studying the evaluation findings and for formulating 
recommendations. The team's involvement in this task is critical in terms of 
developing their sense of ownership of the recommendations and their 
commitment to implementing them. In this phase, the team also formulates 
conclusions regarding the participatory methodology, 
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Pk^se IV: Workshop to formulate lessons learned 



Step 14: Formulate lessons learned for each evaluation 
question 

Step 15: Team assessment of the evaluation process 
Step 16: Summarize evaluation results 
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Step 14: Formulate lessons learned for each evaluation question 



As stated earlier, in a learning process approach to program evaluation, the 
ultimate aim of the evaluation exercise is to develop lessons learned for 
application in the future. In this step, all of the Evaluation Team members 
who participated in the workshop in Phase II come together again for 2-3 
days to discuss the Fieldwork Teams' findings and to develop a set of lessons 
learned. The role of the Evaluation coordinator, as in other phases of the 
process, is both to provide the group with a structure for the task and to 
participate as one of the team members. 



The lessons developed by the team are to be based not only on the problematic 
aspects of the program identified in the evaluation but on the positive or 
effective aspects as well. Important lessons can be learned not only from what 
did not work but also from what worked well and can be used in the future. 



Typically there are a large number of evaluation questions and, therefore, of 
findings. For this reason, the group process must be clearly structured in 
order for the group to accomplish the task in a timely fashion. It is suggested 
that for this task the participants be grouped by program components, in the 
same groups in which they worked in Step 6 (development of evaluation 
questions). In this way, the team members who developed the evaluation 
questions for a particular program component will study the findings and 
develop lessons for that same component. 
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Each of the team members should receive a list of the findings for his/her 
program component. Each group should have a strong facilitator who 
encourages group members to systematically discuss each of the findings and 
to formulate one or more lessons learned. The skill of the facilitator in helping 
the group to develop clear and concise lessons is of utmost importance. The 
clearer the lessons learned the more easily they will be understood by others. 



The GAFNA Experience 



Examples of evaluation findings and lessons learned 



Evaluation Findings 



Development of the nutrition 
themes & messages was based on 
two excellent quantitative- 
qualitative studies on maternal 
dietary beliefs & practices and on 
child feeding practices. 



communicators (TCs) 

The majority of the TCs are 
Mandika speakers and most of the 
training was conducted in 
Mandika. The training was not 
adapted to the language needs of 
the Sarahule and Wolof speakers 
who, therefore, did not fully 
benefit from the training. 



subsidy. 

In most cases, the nurses report 
that they received their fuel 
subsidy (D35 per month). 
However, in many cases, the 
payments were received late out in 
the regions. 



Lessons Learned 



Development of themes and 
messages for nutrition education 
should reflect cultural beliefs and 
practices of tire target group. In- 
depth qualitative studies are 
particularly important as a basis for 
the development of the educational 
content. 



It is not appropriate to conduct a 
training program in only one 
language. If there is ethnic 
variation amongst trainees, the 
training must be adapted to the 
linguistic needs of all. 



The system for distributing fuel 
subsidies should be reviewed to 
ensure timely payment in all 
regions. 
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The involvement of program stakeholders in the development of lessons 
learned in this way has proved to be effective. The lessons which stakeholders 
develop tend to be based not only on the evaluation findings but also on their 
understanding of policy priorities, program context, resource availability, etc. 
Participation fosters ownership. It has been found that where program 
stakeholders have participated in developing lessons, they not only appear to 
have a clearer understanding of the evaluation results and of how they 
should be used, but also a greater commitment to implementing the 
recommendations. The complete set of lessons learned should be included in 
the evaluation report. 

Step 15: Team assessment of the evaluation process 

In this step, feedback is elicited from the entire evaluation team on the 
evaluation process itself. This is an opportunity for all of the team members to 
reflect on the evaluation process and to generate conclusions regarding the 
participatory methodology. Given the innovative nature of the evaluation 
methodology, it is important that it be assessed and lessons generated 
regarding its usefulness in the future. A simple tool can be developed for 
carrying out this assessment. 

The rationale for the use of the participatory methodology is that the 
approach contributes to the development of practical lessons for improving 
program implementation and to the development of staff skills. The 
assessment of the methodology should, therefore, address these two aspects. 

Different techniques can be used to carry out this assessment including: an 
individual written questionnaire; collective assessment in small group 
discussions. The Evaluation Coordinating Group can be responsible for 
designing an assessment tool. 
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In the GAFNA evaluation team members completed an 
individual written questionnaire on their impressions of 
the evaluation process. Here are a few of their responses: 

A forum for discussion 

"The evaluation activity created a forum for those who 
were involved at different levels of program 
implementation to review our combined efforts and to 
decide together how we can improve the program 
strategy." (Josephy Jassey, GAFNA Field Supervisor) 

Strengthening team collaboration 
"The evaluation helped me to understand that every 
individual believes that his/her ideas and thoughts are 
important. By respecting the point of view of other team 
members your relationship with them is strengthened 
and all team members work harder." (Baboucar Jobe, 
GAFNA Project Assistant) 

Learning from others 

"I benefited from the other team members who shared 
their knowledge and experience with me." (Malang 
Fofana, Nutrition Assistant) 

Listening to the community 

"The evaluation created a forum for us to listen to the 
community's views, criticisms and suggestions to ensure 
more realistic and appropriate planning in the future." 
(Muhamed Freeman, GAFNA Field Supervisor) 

Demystifying evaluation 

"Evaluation has ceased to be a word I fear because now I 
know how to plan an evaluation exercise that will produce 
valid results." (Kinday Samba, GAFNA Project Manager) 

Practical lessons for the future 

"The evaluation helped us develop comprehensive and 
practical lessons for future programs." (Saihou Sanyang, 
GAFNA Project Manager) 

Future evaluations 

"I can't overemphasize the need to apply this type of 
participatory methodology in the future to help us 
improve our nutrition programs." (Bintou Keita Kunju, 
Food & Nutrition Unit) 
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Step 16: Summarize evaluation results 



In most cases, both the evaluation findings & lessons learned will be lengthy. 
Therefore, it is useful to prepare a summary of the evaluation results. The 
summary should address the major elements in the program framework. This 
task can be completed by several of the Evaluation Coordinating Group 
members in collaboration with the Evaluation Coordinator. This task will be 
very time consuming if more than 3 or 4 people are involved. 



The GAFNA Experience 



In the GAFNA evaluation summary, results were 
developed for each of the key aspects of the components 
in the project framework. Here are two examples of the 
summary results. 

Project monitoring & supervision : Monitoring and 
supervision of project activities at field level took place 
but was neither systematic nor frequent. This was largely 
due to the fact that monitoring roles, guidelines and 
tools for key actors were not developed. In the future, 
monitoring roles must be defined, tools developed and 
training of key actors provided to ensure efficient and 
systematic monitoring and supervision of activities at all 
levels, including follow-up of mothers in satellite 
villages. 



evaluation team concluded that the most effective 
nutrition education method was the combined use of the 
Traditional Communnicators and Community Health 
Nurses. While the Traditional Communicators are skilled 
at disseminating information in culturally and 
linguistically appropriate ways, the involvement of the 
Community Health Nurses is important to reinforce the 
Traditional Communicators' knowledge of nutrition 
concepts and to help them plan and evaluate their 
activities. 
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Phase V: Development of Action Plan 

In this phase, a draft program action plan is developed based on the evaluation 
findings and lessons learned. This task should be carried out by a small group of 
program managers and implementors, and can take one or two days or more, 
depending upon the level of detail desired. 




f .■ ssssssss Phase V: Development of Action Plan 

Step 17: Develop action plan based on evaluation 
findings 

^ ^ 



Step 17: Develop action plan based on evaluation findings 

In most evaluations, the last step in the evaluation process involves the 
development of recommendations. In the case where the program evaluated 
is ongoing, evaluation results should be directly applied to the program. 
Often, however, there is a lack of continuity between evaluation results and 
their application to program planning. In many cases when the application of 
the findings is left for "later" the application does not take place. Many 
excellent evaluation reports have gathered dust on program managers' 
shelves and were never used. 

To overcome this problem, it is proposed that the evaluation exercise should 
include this step, in which a draft action plan for the program is developed 
based on the evaluation findings and lessons. The product of this step will not 
be a polished, final action plan but should include the main program 
objectives, components, activities and strategies based on the evaluation 
results. Experience has shown that the inclusion of tiiis step as an integral 
part of the evaluation process increases the likelihood that evaluation results 
are integrated into program planning. 

Details of the action planning process are not discussed here as most 
organizations have their own approach to program planning. It is suggested 
that a sub-group of the larger evaluation team be involved in this planning 
exercise. Their draft plan should later be discussed with the larger group. 
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<££&*!£& Phase VI: Finalization and Dissemination of the 
Evaluation Report 

In this important last phase in the evaluation process two important but 
sometimes neglected steps are included. 



>■ — Pha'je VI: Finalization and Dissemination — 

of the Evaluation Report 

Step 18. Write evaluation report 

Step 19. Distribute and discuss evaluation results with 
program collaborators 



Step 18: Write evaluation report 

An obvious, but not always completed step, is the preparation of the 
evaluation report. It is important that at the outset of the evaluation process 
(Phase I) responsibility for writing the report be assigned to one or more 
persons involved in the study and a date for completion decided upon. This 
is frequently overlooked and can contribute to delays in finalizing the report. 

The content and style of an evaluation report depends upon how its function 
is defined. Should it merely report the results or should the steps in the 
process be described? In most evaluation reports, the focus is on reporting the 
findings and recommendations. If, however, the report is viewed as an 
"educational tool" which can help the reader to learn about tire methodology 
itself, then the steps in the process should be discussed as well. In any case, it 
is important that it be written in simple language so that it can be understood 
by all program stakeholders. 

The report should include a 1-2 page "executive summary" of the major 
findings and recommendations. While hopefully the whole report will be 
read by those who receive it, many people will not have time to read the 
entire document. The Executive Summary appears at the beginning of the 
report but it should be written last. 

All of the lessons learned and summary findings prepared in Phase IV should 
be included in the report, as weP as the summary results from step 16. The 
action plan may be voluminous and it should be decided whether to include 
it in the same document or for it to be separate. 
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Step 19: Distribute and discuss evaluation results 
with program collaborators 

An important aspect of any evaluation is that the findings be shared with all 
program collaborators. It is critical that everyone involved in the program not 
only be informed of important lessons learned but also have the opportunity 
to discuss the results. In most cases, only a few copies of an evaluation report 
are produced and hence, it is distributed on a limited scale. Alternatives to the 
typical and limited approach should be explored and the Evaluation 
Coordinating Group should develop a plan for how the findings can be 
distributed and discussed with all key program collaborators. 

Creative strategies for diffusing study results from the central to the 
community level should be explored. A few approaches which have been 
successfully tried include: 

• preparing different versions of the study findings for different readers, 
for example, the full evaluation report can be distributed to a limited 
number of stakeholders; 

• a summary of the report (perhaps 10 pages) can be distributed to all 
program staff who were in any way involved with the evaluation; 

• a 2 -page "evaluation highlights" can be widely distributed to a 
variety of people who would be interested in the main evaluation 
findings and recommendations; 

• holding meetings with program collaborators in each of the program 
regions to discuss the lessons learned and their possible application; 

• asking field workers to discuss the main findings and 
recommendations with program collaborators at the community level. 

There are certainly other creative ways that evaluation results can be 
distributed and discussed with the interested persons. 
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Shortly after the evaluation was completed, GAFNA 
organized a series of meetings with different categories of 
project collaborators from the ministry to the community 
level to discuss the lessons learned and to plan how to put 
them into practice. According to the project manager, 

"The evaluation report became a tool for discussion and 
action rather than being forgotten to gather dust in 
people's offices." 
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Conclusions on m 
the Methodology 



Experience in various countries has shown that a number of the constraints 
associated with traditional evaluation methods can be overcome through the use 
of the participatory methodology described in this manual. There are a number 
of key features of the methodology which appear to contribute to its 
effectiveness. 



Involving program stakeholders 

By involving program stakeholders (information-users and decision- 
makers) in all phases of evaluation planning and implementation they can 
overcome their anxiety about evaluations and create a sense of ownership 
of evaluation results. 

Simple data collection techniques 

Through the use of simple data collection and analysis techniques all 
program staff can be actively involved and can develop basic data 
collection skills. 

Focus on lessons learned 

By focusing the evaluation exercise on developing the lessons learned 
from program implementation, program stakeholders can more openly 
analyze past problems and successes. 

Strengthening team collaboration 

The involvement of different levels of project collaborators as team 
members can contribute to the development of more open relationships 
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between them during the evaluation activity which often continue after 
the evaluation exercise. 

Combining outside and inside evaluators 

The outside evaluator brings objectivity, skills in data collection methods 
and in group facilitation. He/ she orchestrates the process and participates 
as a team member. The insiders bring to the evaluation their intimate 
knowledge of the program and their commitment to discovering how to 
improve their program. 

Practical recommendations for the future 

Based on evaluation findings, team members develop concise and 
practical, recommendations related to each of the program components. 
The involvement of program staff helps assure that the recommendations 
are realistic and feasible. 

Revised action plan 

The evaluation does not end with the formulation of beautiful recom- 
mendations. A last step in the process is the development of a revised 
program action plan which incorporates the lessons learned in the 
evaluation. 

Learning experience for program stakeholders 

Past evaluation team participants agree that involvement in the entire 
evaluation process increases both their understanding of program 
components and dynamics and also their skills in program evaluation. 

Learning experience for evaluation coordinator 

Each participatory evaluation is an occasion for the evaluation coordinator 
to explore alternative ways of structuring the evaluation process and of 
strengthening participant learning. 
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Caveats of c 
the Methodology O 



The participatory evaluation methodology presented in this manual has been 
used with health and nutrition programs in a variety of settings. It has been 
shown to be an effective tool for involving program stakeholders in the 
evaluation of their own programs and for developing practical lessons for 
improving program performance. However, the process is not magic. The success 
of a participatory program evaluation depends on a variety of factors related to 
the availability and carefully structured use of human and other resources. A 
number of caveats, ail discussed in detail in this manual, must be seriously taken 
into consideration if the methodology is to be successfully used. 



Careful planning 

Each step in the evaluation process must be carefully planned with respect 
to the timing, human and material resourc o, = required and responsibility 
for the tasks at each step in the process. To facilitate the planning process, 
evaluation organizers should start by reading the entire manual so that 
they have a clear idea of what is involved from start to finish. Only then 
will they be able to begin planning in detail how to organize the process. 

Coordination of fieldwork lor istics 

Perhaps the most complicated aspect of the entire evaluation process is the 
planning required for the fieldwork phase. Logistical planning for the 
fieldwork must begin early and responsibility for logistical arrangements 
before and during the fieldwork clearly defined. 
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Scheduling data collection and analysis 

In developing the daily schedule for fieldwork data collection, it must be 
remembered that the analysis of qualitative data is very time-consuming. 
As a rule of thumb, the analysis of a set of interview data will take 
approximately twice as long as conducting the interview itself. It is 
recommended that the time planned for actual data collection, including 
travel time, be limited to a half-day to leave the remainder of the day for 
data analysis. 

Skills in qualitative data collection and analysis 

Most teams have reported that the most difficult part of the entire 
evaluation process is the analysis of the qualitative data. It is imperative 
that at least one person on the evaluation team, and preferably several, 
have strong skills in qualitative data collection and analysis. The 
evaluation coordinator should have these skills. A team of persons with 
skills only in quantitative research methods will encounter numerous 
problems using this essentially qualitative methodology. 

Involvement of program managers 

A tendency has been observed in participatory evaluations for program 
managers to delegate too much responsibility to program implementors 
for conducting the evaluation. The concept of stakeholder involvement in 
the evaluation implies that both program implementors and managers 
must be actively involved in all phases of the process. The involvement of 
program managers is of critical Importance to ensure that their priorities 
and opinions are incorporated into the evaluation process. 

Effective group facilitators 

For all of the group planning, training, and data analysis sessions 
experienced group facilitators are required. Group facilitators should have 
skills in active listening, questioning and synthesizing. The presence of 
good facilitation skills will help ensure that groups accomplish their tasks 
in an effective and timely fashion. 

Scope of data collection 

The evaluation methodology is particularly suited for use with 
community-based programs. However, an evaluation of such programs 
should include data collection not only at the community level, but also at 
the institutional level from staff who are involved with the program. It is 
only by asses= ; ng the opinions and attitudes of institutional and 
community actors that the evaluation will yield a holistic understanding 
of program accomplishments and constraints. 
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Experienced fieldwork team leaders 

The success of the fieldwork phase depends largely on the skills of the 
team leaders. They must be carefully chosen in terms of their: physical 
stamina; ability to motivate and coordinate team work; group facilitation 
skills; analytical thinking skills required for qualitative data analysis; 
experience with the type of program being evaluated. 

Time commitment on part of team members 

The evaluation methodology involves a learning process which can be 
effective only if team members fully participate in that process. This 
requires that individuals chosen as members of the full evaluation team 
and/or of the field teams must be available to participate on a full-time 
basis for the duration of the corresponding phases of the evaluation. 

Support from management 

The use of the participatory methodology will be effective only if it is 
understood and supported by management. Institutional and program 
managers must understand the rationale and practicalities of this type of 
methodology and must provide the human and other resources required. 
Without their full support the evaluation cannot succeed. 

Choice of evaluation team members 

Both the effectiveness of the evaluation process and the later impact of 
evaluation results depend upon choosing carefully the persons to be 
involved in the process. Choice of the program stakeholders, the decision- 
makers and potential information-users, must be carefully made to ensure 
that the right people are included in the process. 

The challenge 

The methodology proposed in this manual may seem like a rigid recipe. It 
is hoped, rather, that it will be seen as a framework which helps groups of 
persons within programs and institutions to design participatory program 
evaluations using some of these elements and modifying others. 
Participatory evaluation is a young field and the best ways to go about it 
remain to be discovered. The challenge to all of us is to experiment and to 
develop methodologies which help us to more effectively learn from 
program constraints and successes. 
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The following references which are organized by topic areas, would be useful for 
those who want to read more about how to plan and conduct both participatory 
evaluations and other types of community studies. 



Qualitative data collection methods 

Agar, M.H. Speaking of Ethnography. Sage Pub., Beverly Hills, 1986. 

Bogdan, R.C. & S.K. Biklien. Qualitative Research for Education; an 
introduction to theory and methods . Allyn & Bacon, Boston, 1982. 

Patton, M. Q. Qualitative evaluation methods . Sage Pubs., Beverly Hills, 
1980. 

Patton, M.Q. How to use qualitative methods in evaluation. Sage Pubs., 
Newbury Park, 1987. 



Group interview! focus group technique 

Aubei, j. A. Guidelines for p lannin g. a nd co nd ucting stu di es usin&.the group 
interview technique . International Labor Organization, Geneva, 1994. 

Kumar, F. Conducting Group Interviews in Developing Countries. AID 
Program Design and Evaluation Methodology Report No. 8, USAID, 
Washington D.C., 1987. 

Krueger, R.A. Focus Groups: a Practical Guide for Applied Research, Sage. 
Newbury Park, 1988. 
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Morgan, D.L. Focus groups as qualitative research . Sage, Newbury Park, 1988. 



Rapid Assessment Procedures (RAP) and Rapid Rural Appraisal (RRA) 

Chambers R. Rural Development: Putting the Last First . John Wiley & Sons, 
N.Y. 1983. 

Khon Kaen University. Proceedings of the 1985 International Conference on 
Rapid Rural Appraisal. Khon Kaen, Thailand: Rural Systems Research and 
Farming Systems Research Projects, 1987. (Available from: Dr. Terd 
Charoenwatana, Faculty of Agriculture, Khon Kaen University, Khon Kaen 
40002, Thailand. $20.00 for book including airmail postage). 

Schrimshaw S. and E. Hurtado. Rapid assessment procedures for nutrition 
and primary health care. U.N. University, Tokyo/UNICEF/UCLA Latin 
American Center, 1987. (Available from: UCLA Latin American Center, 
University of California, Los Angeles, California 90024.) 



Group dynamics/ group facilitation skills 

Bolton, R. People Skills. Prentice-Hall, Inc. Englewood Cliffs, New Jersey, 
1979. 

Miles, M.B. Learning to Work in Groups: A Practical Guide for Members and 
Trainers. Teachers College, Columbia Univ., New York, 1981. 

Vella, J. Learning to Listen: A guide to adult nonformal education. Center for 
International Education, Univ. of Massachusetts, Amherst, 1979. 



Participatory evaluationlprogram evaluation 

American Council of Voluntary Agencies for Foreign Service. Evaluation 
Soureebookfor Private and Voluntary Organizations. New York, 1983. 

PfohL Jacob. Participatory Evaluation: A Users Guide . Private Agencies 
Collaborating Together (PACT), 777 U.N. Plaza, New York, N.Y. 1986. 

Fuerstein, Marie Therese. Partners in Evaluation. MacMillan Education, 
London, U.K. 1986. 

Patton, M.Q. Creative Evaluation . Sage Pubs. Beverly Hills 1981. 

Rugh, Jim. Se lf-Evaluation: Ideas for Participatory Evaluation of Rural 
Community Development Projects. World Neighbors, 5116 N. Protland Ave., 
Oklahoma City, Oklahoma, 1986. 



